Skip to content

Reduce event loop saturation for scan throughput#3019

Merged
liquidsec merged 7 commits intoblasthttp-integration-cleanfrom
eventloop-optimization
Apr 3, 2026
Merged

Reduce event loop saturation for scan throughput#3019
liquidsec merged 7 commits intoblasthttp-integration-cleanfrom
eventloop-optimization

Conversation

@liquidsec
Copy link
Copy Markdown
Contributor

@liquidsec liquidsec commented Apr 2, 2026

Summary

  • Offload CPU-bound work (DeepDiff, xmltodict, ML prediction) from event loop to run_in_executor_cpu
  • Reduce module thread counts (paramminer 12→4, iis_shortnames 8→4, excavate 8→6, badsecrets capped at 4)
  • Replace request_batch/request_custom_batch task-per-URL pattern with fixed worker pool
  • Remove unnecessary async lock acquisitions from module worker pre/postchecks
  • Binary search for paramminer count_test instead of linear probe
  • Single mid-scan canary check per extension in web_brute instead of per-hit

Results (same scan, same targets)

Metric Before After
paramminer loop share 43.0% 16.5%
Peak concurrent tasks 579 317
URLs produced 237 (89 min) 1,059 (63 min)
cloudcheck backlog 15,015 gone

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 2, 2026 •
edited
Loading

📊 Performance Benchmark Report

Comparing blasthttp-integration-clean (baseline) vs eventloop-optimization (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name 📏 Base 📏 Current 📈 Change 🎯 Status
Bloom Filter Dns Mutation Tracking Performance 4.16ms 4.36ms +4.8% ⚪ ✅
Bloom Filter Large Scale Dns Brute Force 17.47ms 17.37ms -0.6% ⚪ ✅
Large Closest Match Lookup 353.40ms 353.67ms +0.1% ⚪ ✅
Realistic Closest Match Workload 190.20ms 188.42ms -0.9% ⚪ ✅
Event Memory Medium Scan 1779 B/event 1779 B/event +0.0% ⚪ ✅
Event Memory Large Scan 1768 B/event 1768 B/event +0.0% ⚪ ✅
Event Validation Full Scan Startup Small Batch 424.66ms 407.94ms -3.9% ⚪ ✅
Event Validation Full Scan Startup Large Batch 587.42ms 581.72ms -1.0% ⚪ ✅
Make Event Autodetection Small 31.11ms 30.67ms -1.4% ⚪ ✅
Make Event Autodetection Large 313.17ms 314.01ms +0.3% ⚪ ✅
Make Event Explicit Types 14.03ms 13.82ms -1.5% ⚪ ✅
Excavate Single Thread Small 4.002s 3.964s -0.9% ⚪ ✅
Excavate Single Thread Large 9.819s 9.511s -3.1% ⚪ ✅
Excavate Parallel Tasks Small 4.167s 4.232s +1.6% ⚪ ✅
Excavate Parallel Tasks Large 6.590s 6.681s +1.4% ⚪ ✅
Is Ip Performance 3.18ms 3.17ms -0.5% ⚪ ✅
Make Ip Type Performance 11.48ms 11.57ms +0.8% ⚪ ✅
Mixed Ip Operations 4.56ms 4.51ms -1.2% ⚪ ✅
Memory Use Web Crawl 163.1 MB 168.4 MB +3.2% ⚪ ✅
Memory Use Subdomain Enum 19.3 MB 19.4 MB +0.3% ⚪ ✅
Scan Throughput 100 4.557s 4.615s +1.3% ⚪ ✅
Scan Throughput 1000 38.188s 36.074s -5.5% ⚪ ✅
Typical Queue Shuffle 64.37µs 65.62µs +1.9% ⚪ ✅
Priority Queue Shuffle 725.38µs 729.12µs +0.5% ⚪ ✅

🎯 Performance Summary

✅ No significant performance changes detected (all changes <10%)

🐍 Python Version 3.11.15

@codecov
Copy link
Copy Markdown

codecov bot commented Apr 2, 2026 •
edited
Loading

Codecov Report

❌ Patch coverage is 92.52336% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 91%. Comparing base (b19ff4b) to head (0d68bcc).

Files with missing lines Patch % Lines
bbot/core/helpers/web/web.py 94% 3 Missing ⚠️
bbot/modules/web_brute.py 75% 3 Missing ⚠️
bbot/modules/paramminer_headers.py 88% 2 Missing ⚠️
Additional details and impacted files 
@@                     Coverage Diff                     @@
##           blasthttp-integration-clean   #3019   +/-   ##
===========================================================
- Coverage                           91%     91%   -0%     
===========================================================
  Files                              443     443           
  Lines                            37861   37835   -26     
===========================================================
- Hits                             34281   34237   -44     
- Misses                            3580    3598   +18

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@liquidsec liquidsec merged commit 0317823 into blasthttp-integration-clean Apr 3, 2026
4 of 16 checks passed
@liquidsec liquidsec deleted the eventloop-optimization branch April 3, 2026 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@liquidsec